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ABSTRACT 

This paper describes the development and 
implementation of an evaluation system for a pediatric residency 
program to meet federal grant monitoring requirements. Continuous 
communication and collaboration with program participants 
characterized this naturalistic formative evaluation. Concerns 
identified through initial interviews of the 15 residents and six 
faculty members were formulated into evaluation goals in six areas: 
(1) the ambulatory setting; (,2) the inpatie-'t setting; (3) 
subspecialty support; (4) community involvement; (5) resident 
evaluation; and (6) teaching evaluation. Focusing on general goals 
rather than individual performance, evaluation activities to collect 
both qualitative and quantitative data were chosen for their accuracy 
and feasibility in collaboration with the faculty. This participatory 
model provided both an effective means of evaluation and a systematic 
framework for program change and improvement by faculty, residents 
and staff. (BS) 
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A Fart ici pant Model lot Evaluatlog a Prixa^cy Care Resideacy Program 

- This paper describes the developme iC aad impIacaeaCatioa of a system for 
evaluating a primary care residency program in pediatrics « The effort was 
instituted because of requirements of a federal graot which demanded monitoring 
oz liie residency program. The evaluation design was formative and characterized 
by early and continued involvement of the faculty and residents* 

r 

Background 

Many of the traditional methods were not practical for the evaluation of 
this pediatric training program. Those models which viewed evaluation as a 
technology for applying measurement science were too restrictive* Any component 
of the program (e*g*» the ambulatory settings individual resident instruction^ 
faculty teaching) could have provided a basis for a pretest/posttest sttidy* 
However^ the goal of the study required that the entira educational program be 
monitored; designs to evaluate program components were insufficient to achieve 
this type of result* 

The approach we developed was compatible with the naturalistic evaluation 
models offered by Lincoln and Cuba (1981), Stake (1981), Parlett (1977), and 
Rippey (1973)* We wanted to present qualitative and quantitative information to 
the participants which would provide a basis for change within the prog4:am* Ve 
first described the program, att^pting to understand how it functioned, and 
pointed out areas of weakness .and strength. As the evaltiat ion system developed 
and as new areas of investigation were pursued, information was continuously fed 
back Into the program • The evaluators became a part, albeit an Independent 
part, of the program itself. 



Methods 

The pediatric residency was a primary care training program consisting of 
15 residents and six full-time faculty members • The evaluation process was 
begun by interviewing faculty, residents, and staff » The format of these 30 to 
45 minute sessions was loosely structured, but always included the following 
questions: (1) What are the strengths of the program? (2) What are the 
weaknesses of the program? (3) How might the evaluation process help you, i.e,., 
what type of infcrraation would help you to Improve the program and its 
functioning on a dally basis? 

Axialysis of the interview data yielded 14 consistent concerns* Thei*e were 
presented to th ! faculty for written reactions and views on the relative 
importance of each*. By analyzing the reactions, we were able to approach the 
original 14 concerns with the following six evaluation goals: 

1. To determine if the ambulatory setting provided an appropriate model 
of group practice and an appropriate base for teaching outpatient 
pediatrics. 
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2. To detenaine If the connnunity hospital setting provided aa 
appropriate baae for teachiag Inpatient pediatrics, and to compare 
the inpatient population to that of a university hospital. 

3. To determine subspecialty support la the hospital setting by 
docuaenting the number of subspecialty encounters; to determine 

\ subspecialty support in the ambulatory clinic by documentiog 

the characteristics of the subspecialty clinics (number, 
type, attendance, patient profiles); to docunen!: available 
elective rotations. 

A. To document resident participation in the community-oriented 
parts of the residency program. 

5. To develop resident evaluation techniques acceptable to the faculty 
and house staff » 

6» To develop a systematic method for determining the quality of the 
teaching process. 

The evaluation goals were not identical to the educational gc^ls of the 
pediatric training program. The educational goals were a broad outline of an 
ideal training program in primary care pediatrics. They encompassed ©any areas, 
from accurate diagnosis of medical problems to identification of commxiaity 
problems which affect the health of children. The evaluation goals focused on 
areas which dealt primarily with resident education and teaching. (Had we 
focused our evaluation efforts on the stated program goals, we would have been 
overwhelmed with the magnitude of the project and may have focused on data 
collection in areas which were not directly related to resident education and 
teaching.) The six evaluation goals reflected the primary concerns of all those 
involved in the program and provided the basis for our project over a four-year 
period. 

Once our goals were defined, they were matched with specific evaluation 
activities. At a meeting, tae faculty was presented with the six evaluation 
goals along with tentative evaluation activities. Final activities were chosen 
by considering the accuracy of the data which might be obtaitied and the 
feasibility of implementation. This collaboration led to decisions matching 
goals with appropriate activities. These are shown In Table 1, 



erJc 



Results 

Evolution of this evaluation system had three results: (1) the impact of 
each evaluation activity, (2) cumulative effects of all activities, and (3) 
insight into this system of evaluation. 

Results of the Evaluation Activities 

Among the most valuable of all the activities and one which explored every 
goal, was a series of periodic, confidential interviews with the faculty, staff, 
and residents. These occurred at six month Intervals, and were structured to 
include questions about each person's recent experiences. These Interviews made 
it possible to obtain detailed opinions about specific program components. Ihe 
major concerns from the interviews were abstracted and shared among the faculty. 



BEST COPY AVAIUBLE 



page 3 



staff, /iud residents ailowlug qualitative assessment of most aspects of the 
pcograo. 

Frosi the ab^itract^d latervlew sua tier iai, ve CQopiled a list of changes which 
had been suggested la the interviews. The list was divided into three sections: 
(1) those changes which would require a major organizational effort, e^g^ the 
development of an adolescent medicine cliaic and rotation, (2) liiaitGd changes 
or asaignaents requiring an individual, usually a faculty menber, to execute 
them, e#g«, writing a list of goals and objectives for a patient education 
group, and (3) Those changes which needed only a place in the schedule, e«g», 
resident presentation from their elective rotations* This list of specific 
changes from an outside objective source promoted change within the program. 

Goal #l! Ambulatory Setting s In the ambulatory setting, long tens 
doctor-patient continuity was examined by tracking newborns and their primary 
caregiver from an initial visit at two weeks of age through their two year 
check-up. It was possible to determine the overall continuity for the entire 
clinic, individual residents and group practices. 

The telephone survey of patient satisfaction was done by recruiting 
patients who appeared at the outpatient facility over a three-day period, and 
then randomly selecting participants from this sample* Over 90Z of the people 
polled indicated complete satisfaction with the care received, so this activity 
was not repeated* 

A survey of end^f-^t he-day summaries of patient visits allowed analysis of 
presenting diagnoses* These summaries over several weeks substantiated the bigh 
number of patients with primary care probl^s. 

Goal 02: Inpatient Setting , On the inpatient setting, the scores on the 
American Board of Pediatrics In-Trainir.^ Exam provided data on individual and 
cumulative resident performance. Anal/, s by subspecialties provided 
information on areas of strength and weakness, and afforded a basis for 
adjusting the curriculum* The analysis of discharge diagnoses provided data on 
the reason for hospital admissions* Like the end-of--the-day summaries In the 
ambulatory setting, this activity documented the programme ability to provide 
adequate numbers of primary care patients* Comparison of these discharge 
diagnoses with those from a university hospital, where the numbers of tertiary 
medical problems were high, demonstrated the primary care emphasis of the 
inpatient setting* 

Goal ^3: Subspecialty Support s In the area of subspecialty support, 
analysis of the In-training Exam by subspecialty provided information on 
strengths and weaknesses of the program and individual resident performance* 
This allowed adjustments in the program where all the residents demonstrated 
weakness, and allowed individuals to change their elective rotations and 
outpatient experience to address personal deficiencies* 

The documentation of clinical conferences and si:ibspecialty clinics was a 
quantitative exercise which substantiated that all important subspecialty areas 
were adequately covered by didactic sessions or outpatient clinics* The 
documentation of calls to subspecialists was impossible to complete because it 
relied heavily on continual logging of telephone calls by the resident staff. 

Goal Community Involvement* The interviews assessed resident opinions 
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Oft a variaty of problems related to medical, social, and behavioral issues 
within the coraaunity educational experience. In addition, two different 
questlunuaiiTes were used to gauge residents' attitudes and opiaions about 
participation ia the community pro^jram. Direct observation during the school 
rotation allowed description of this program and its effect on residents and the 
schools involved. 

Goal 05: Reside nt Evaluation . Resident e valuation was approached fron 
several directions, la addition to the In-Training Exaa, which was used to 
assess each resident's overall and subspecialty perfonaance, a monthly 
evaluation form was completed by a faculty oemuer who had significant 
educational contact with each resident. With written comments from the end of 
these forms, a summary of the year's evaluations was prepared for each resident. 
Chart audits, tuo each on the inpatient and outpatient service, were required. 
This system of audit was designed to assess the abilty of the resident to 

i synthesize the history and physical exam into a problem list. 

j 

Goal #6; Evaluation of Teaching and Curriculum . Evaluation of teaching 
and curriculum was done in three ways. A questionnaire was sent to graduates 
and proved a wonderful impetus for program change. The questionnaire was 
comprehensive, covering all the components of the curriculum and all the 
possible subspecialty areas. Ample space for comments was provided, and most of 
the graduates took advantage of this to explain the reasons for their answers. 
Tlie results of this questionnaire showed that some areas which were thought t;> 
be only marginally important (e.g., minor suturing and surgery, outpatient 
orthopedics, dental medicine) were an important component of everyday 
pediatrics. Although it is not possible to cover all areas In depth in a three 
year training program, this information suggested more relevant elective and 
conference topics in a program committed to primary care. 

Yearly evaluations were completed by the residents for each faculty member . 
Tlie forms were completed confidentially and resident anonymity was guaranteed. 
The copsnents on the form were summarized. K . le of the faculty had ever 
participated in a program where their teacl-'ag was assessed. Initially, this 
was very difficult, but from subsesquent Interviews and evaluations it was 
possible to determine that the faculty had made changes In their teaching 
styles. The resident's perception of the quality of teaching and the 
availability of the faculty improved from year to year. 

A curriculum questionnaire was completed by all the residents at the end of 
each year. Using a scale of one to four, the residents rated all aspects of 
their curriculum, faculty involvement, and clinical supervision. Quantifiable 
data was obtained on all aspects of the program. 

Cumulative Effects 

Perhaps the most important result of the evaluation process was the 
cumulative effect of the separate evaluation activities. Before the evaluation 
began, any impetus for change was on an ad hoc basis, if it ocurred at all. The 
changes that did occur began because of intense resident or faculty 
dissatisfaction. Our evaluation process allowed a systematic and thorough 
procedure for Identifying and exploring problems from multiple perspectives. 
The program became more open, and less resistant to change. 

Evaluation Results 
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There were results for the evaluatioa process itsalf . By approaching 
evaluation with aa open nilnd and by gaining <-ooperatlon from all sides, It was 
possible to develop a systeqt whicn was effective without belag Judgnental or 
threatening* The evaluatioa effort vas done with ths prograa parcicipaats, not: 
to them or for them. It was possible for all to keep their autoaomy, but work 
in mutually constructive ways. We were able to put a lot of poteat, 
emotlonally-ladea iaformation into the hands of the prograa participants without 
destroying personal or professional relationships. In addition, the evaluatioa 
effort enhanced conraunication among participants setting a precedent foe faculty 
retreats and formal faculty-resident conferences where grievences and new ideas 
were discussed* 



Discussion 

Hie evaluatioa process evolved continuously. As the project developed, 
some Issues persisted, some emerged, some faded. Our dscisioa was to 
concentrate on areas where change and improvonent ««re nose likely. For 
example, some aspects of the p >gram seemed destined to reai< in stable from year 
to year (such as the high level of patient satisfaction or the low percentage of 
long term resident-patient continuity). All agreed that continued efforts in 
these areas would be unproductive. Moreover, some evaluation activities could 
not be Implemented. For instance, residents found it iiapossible to record 
consistently all their calls to subspecialist on the inpatient service, so this 
activity was discontinued with faculty approval. The faculty reaalned actively 
involved in decisions to abandon or develop lines of inquiry. 

We used many different approaches to data collection attempting to strike a 
balance between quantitative and qualitative processes. Quantitative data were 
Important in determining the basic structure of the prograa, qualitative in 
assessing faculty and resident perceptions about the prograa's content and its 
functioning. The two methods of data collection allowed us to crosscheck the 
reliabilty of each as the evaluation process developed. 

Often qualitative data provided more insight than the quantitative studies. 
The interviews uncovered specific concerns about teaching which often resulted 
in changes. For example, one series of interviews highllgbtsd the problea 
caused by a lack of specific rules regarding attendii^ rounds* Faculty and 
residents did not know what was expected of them or of each other. Residents 
complained that many attending rounds became lectures on specialty areas and did 
not relate to patients on the floor. On the other hand, the attending 
physicians felt this was the time to teacl^ their specialty because there were 
other opportunities for the residents to learn inpatient pediatrics. The 
attendings and the residents had totally different expectations. Appreciation 
of this conflict resulted in the creation of a manual outlining the 
responsibilities not only of attending rounds but also of each level of resident 
training and each faculty position. The following year we documented the 
faculty and resident reactions to the responsibilities outlined in that manual. 

This approach to evaluation was designed to assess general goals rather 
than specific areas of individual performance. For example, we were able to 
document the failure of the program to include adequate teaching of emergency 
pediatric procedures. After establishing a monthly conferfpce, we concluded 
that residents were generally more confident in their abilities to handle these 
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emergencies • iMeasurei&aaC: of :;peci£lc co&pcCtMiciuii was beyond our resources, 
although this evaluatioa model does not preclude iacorporacXoa of such 
taeasureoents • 

The evaluators performed dual roles* We attempted to become knowledgeable 
and intiuentlal Insiders , and at the same time we were careful Co preserve our 
Independence » CK^er a relatively short period of tlffle, we gained the confidence 
of the residents, faculty, and staff* We learned to understand the subtleties 
of the program which explained how and why the program funccionad as ic did* We 
could stand back, verify Information, and present It in ways that were not 
threatening and allo^d changes to occur without animosity « 

An ac^vantage of this type of evaluation is that it can t>e carried out with 
limited resources* During this four year project, an evaluation director worked 
lOZ time and a research assistant worked 5OZ time* Vltb this limited staff, ve 
provided the program with descriptive and quantifiable data which were valuable 
in identifying strengths and weaknesses, and providing a basis for change* 



Conclusion 

The essence of our approach was communication. The faculty and residents 
were Involved at all times* This collaboration was important because it 
directed the effort toward areas where changes were most likely. We facilitated 
interactions among program particii^nts, providing a systematic and Impartial 
source of data about specific problem areas* This approach to evaluation made 
the participants realize that there were ways of instituting change when 
weaknesses were identified* Conversely, when streimtbs were documented, they 
provide a base on which the program could develop. 

This participatory model provided both an effective means of program 
evaluation and a framework within which faculty, residents, and staff could 
shape their program* We believe this approach can be generalized beyond 
residency training into other areas, such as nurses* training or an elementary 
school system* Three results of this type of program evaluation^lnvol^rament of 
program participants, continuous change for the better j, and facilitated 
communication-^Hwould be welcome in a variety of settings* 
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* .ble 1: Evalaatioa Goals atid Activities 



Coal Number 
1 

Ambulatory 
Setting 

2 

Inpatient 
Setting 



3 

Subspecialty 
Support 



4 

Community 
iQvolvment 



/ ^ 

/ Resident 

Evaluation 



6 

Teaching 
Evaluation 



Activities 

A« Analysis of long tera resideat-patieut coatlouity 

Satisfaction survey of a random sample of patients 
C* A survey of r«iasoas for patient visits 

A« Analysis of general pediatric knowledge using tha 
American &>ard of Pediatries Xa^Tralning Exam 

B* Comparison of discharge diagnosis with a 
university*->based pediatric residency 
training program 

A« Analysis of the American Board of pediatrics 

In-Training Exam by subspecialty 
B. Clinical conference » elective, and subspecialty 

clinic documentation 
C* Interviews with residents on elective and 

subspecialty exposure 

A. Itesident questionnaire : attitudes and opinions 
on medical and behavioral«developmental issues 

B. Resident questionnaire: school rotation 

C. Direct observation of school and community rotation 
0. Resident evaluation of neurodevelopnenCal rotation- 

A. American Board of Pediatrics lU'-tralning Exami 
Analysis 

B. Monthly Resident Evaluation by Faculty 

C. Chart Audits 

A» Yearly faculty evaluation by residents 

B. Survey of graduates regarding program 

C* Resident questionnaire: curriculum evaluation 
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